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What is Predictive Risk Modeling? 


• Use data already collected about each child to evaluate the 
possibility of future adverse outcomes 

• Why consider predictive risk modeling? 

- Wider availability of high quality information about cross-sector 
service interactions that can inform our understanding of risk and 
protective factors 

- Advances in technology and analytic capabilities...we can use these 
data in real time rather than retrospective analyses 

- Growing appreciation that clinicians are often poor at weighting 
complex factors (and time is scarce!) 
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OPPORTUNITY#!: 

Improving Child Welfare 
Decision Making 


OPPORTUNITY #2: 

Rethinking Prevention of 
Child Abuse &Neglect 


Screening Decisions & Outcomes 


r O 53% ( 12 , 894 ) 

Screened Out 




e 


24,188 

Index Referrals 


-047% ( 11 , 214 )- 

Screened In 


- © 35% ( 4 , 467 ) 

Re-referred within a year 


--©65% ( 8 , 418 ) 

No Re-referrals 


r -©24% ( 2 , 681 ) 

Re-referred within a year 

\ 


l -©76% ( 8 , 533 ) 

No Re-referrals 
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Today: Using Integrated Data 
to Inform Decision-Making 




In Allegheny County, rich data are 
available to case workers to help 
inform initial maltreatment 
screening decisions at the child 
protection hotline, but 

- No standardized protocols for using 
these data to make referral screening 
decisions 

- No method for systematically weighting 
this information in an equitable manner 
across all referrals 

- No understanding of what information is 
correlated / predicts future adverse 
outcomes for children 
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Developi 


• The system will harvest data from 

the data warehouse* 

• At each call, a screening score will 
be produced for each child 
associated with an allegation of 
maltreatment 

• This screening score is comprised of 
the risk of re-referral, given screen- 
out and the risk of placement in 
foster care, given screen-in 

• More than 100 pieces of 
information are included in the 
screening model 


* Only if on MCI_ID is successfully established 
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Developing a Screening Score 


• The screening score is from 1 to 20 

• The higher the score, the higher the chance of the future event 

(e.g., abuse, placement, re-referral) according to the data 



4 5 6 7 8 9 


1 in 20 

calls receive 

a score of 1 

(bottom 5 % of risk) 




i 

1 in 20 

calls receive 

a score of 20 

(top 5% of risk) 
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Researchers built a screening model based 
on information that we already collect 

They identified more than 100 factors 

that predict future referral or placement 

To test if the model might improve the 
accuracy of screening decisions, we scored 
thousands of historical maltreatment calls 
and then followed the children in 
subsequent referrals to see how often the 
model was correct... 



The 



1 in 10 children 


with a score of 1 were re-referred 

within two years of the call 
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9 in 10 children 


with a score of 20 were re-referred 

within two years of the call. 
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The Results: 

Out-of-Home Placements 





1 in 100 children 

who received a score of 1 were placed 
out-of-home within 2 years of the call 
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Placements 



1 in 2 children 


who received 

a score of 20 were placed 
out-of-home within 
2 years of the call 
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Under current practice: 


27% of highest risk cases 
were screened out — 

of these, 1 in 3 are re-referred 
and placed within 2 years of the 
initial screened out call 

48% of lowest risk cases 
were screened in — 

and yet only 1.4% of those 
are placed within 2 years. 
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OPPORTUNITY#!: 

Improving Child Welfare 
Decision Making 


OPPORTUNITY #2: 

Rethinking Prevention of 
Child Abuse & Neglect 


How well do our child serving systems 

choose the right child at the right time? 


Not very well: 

4 in 5 children in this county who died (or nearly died) 
as a result of abuse were never referred to child welfare 

before the incident. 




Generating 


As soon as the birth is registered 
we could assign a needs score 
between 1 and 20 


Predicting a child protection 
case opening by age 3 

- Vision would be to prioritize 
high needs births for upstream 
early intervention support in the 
hopes of preventing the need 
for later child protection 
involvement 
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Generating a Score at Birth 


Of those who 
received a 
risk score of 20, 
40% of them 
resulted in 
an open case 
by age 3 
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How do we know the model is 
identifying truly high-risk children? 

We looked at fatalities and 
near fatalities that were reviewed 
under Act 33 

There were 41 children who 
were victims of maltreatment- 
related fatalities or near fatalities 
in the researched cohort (2007-2014) 

We were able to match 30 children ... 



Identifying Truly High-Need Children 


Of those 30 children, 12 are identified by our model as having 
been highest need at birth. 


Only 6 of these children were ever 
referred to child protective services 









50 % 
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Opportunities for Prevention 


Offer voluntary services at the time of birth 

Use needs score to prioritize home visiting services through 
coordinated intake 

Use needs score to provide extra support to familes who engage 
at a family support center 

Proactively reach out to high-risk families who live in a 
catchment area for family support centers 

Build needs score into screening at Children's Hospital 
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OPPORTUNITY#!: 

Improving Child Welfare 
Decision Making 


A Closer Look 


Process Evaluation 


— Assess implementation process 
— Impact of screening scores on decision making 
— Key questions: 

• How has the tool modified the experience of call 
screeners? 

• What are the practice and policy implications? 

• What are perceptions and reactions to the tool? 

• How extensive were preparations for development of 
the model? 

— Primary methods: interviews, surveys, document review 
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Impact Evaluation 


Impact on: 

- Accuracy of decisions 

— Reduction in unwarranted variation in decision-making 

- Reduction in disparities 

— Overall referral rates and workload 

Outcomes assessed 

- Rate of screen in 

- Likelihood of "bad" screen-outs 

- Likelihood of unnecessary screen-ins 

— Unwarranted variation in screening decisions 

- Disparity in screening decision 


April 12, 2016 


Allegheny County Department of Human Services 


24 



Questions 
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Next Steps 


• Complete peer review 

• Validation with Children's Hospital data 

• Finalize visualizations 

• Training 

• Looking at resiliency/positive deviance 

• Go-live July 1, 2016 

• Evaluate 

• Build 2.0 model, likely including likelihood of critical incident 

• Continue exploring the "preventive model" 
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How this works 
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Calls come into the child welfare 
ca ll screening unrt far processing, 
and a decision on next steps is 
made. 


Example cal I: 

Anonymous call is received fram 
PA Chi Id Line 

2 children (male aged 10, female 
agec 7} living with father and 
stepmother an weekends. 
Allegation states that stepmother 
"shoots heroin" and "leaves them 
alone to wander.' Father sells 
marijuana and uses cocaine. 


Client History Timelines 

(Currently In Development) 


For individuals linked to the incoming 
allegations, a visualization tool will comb the 
Data Warehouse for prior service 
involvement and display it to the caM 
screening worker. 
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Predictive Risk Score 

(Currently In Development) 


As an additional piece of contest, a 
predictive lisk score will be 
calculated to assess whether the 
allegation data, along with other 
known risk factors for the family, 
suggest future adverse outcomes 
relevant to the children's welfare. 


A "thermometer*' icon Me the one Co tfw 
ieff is a type of display option that couid 
be employed to portray the family's risk. 

display options ore sfi'iJ in 
development}. 




Referral Progression 


Child Welfare Allegation: Call Screening 

Call information 
received and 
processed 

limelines. Risk 
Score, and other 
reseanch/context 
considered 

Consultation 
with call 
screening 
supervisor 
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